Unsupervised anomaly detection using Anomalib library¶

Author: Dennis Hernando NÚÑEZ FERNÁNDEZ
Personal website: https://dennishnf.com

Pre-installation settings¶

All the experiments were runned on a machine with Ubuntu 20.04 LTS and NVIDIA GeForce GTX 1050 Ti with 4GB.
In addition, this notebook was locally implemented using Conda and in a virtual environment.
Therefore, to create a virtual enviroment in Conda, the following commands were introduced in the terminal:

$ yes | conda create -n anomalib_env python=3.8
$ conda activate anomalib_env

Installing AnomaLib¶

In [1]:
PROJECT_PATH = '/home/dennishnf/Desktop/unsupervised-anomaly-detection'
In [2]:
%cd {PROJECT_PATH}

/home/dennishnf/Desktop/unsupervised-anomaly-detection
In [3]:
!ls

dataset    unsupervised-anomaly-detection.html
README.md  unsupervised-anomaly-detection.ipynb
In [4]:
!git clone https://github.com/openvinotoolkit/anomalib.git

Cloning into 'anomalib'...
remote: Enumerating objects: 21905, done.
remote: Counting objects: 100% (3444/3444), done.
remote: Compressing objects: 100% (1201/1201), done.
remote: Total 21905 (delta 1974), reused 3122 (delta 1748), pack-reused 18461
Receiving objects: 100% (21905/21905), 49.53 MiB | 4.14 MiB/s, done.
Resolving deltas: 100% (12846/12846), done.
In [5]:
%cd anomalib

/home/dennishnf/Desktop/unsupervised-anomaly-detection/anomalib
In [6]:
!ls

anomalib	    CONTRIBUTING.md  markdownlint.rb  setup.py
CHANGELOG.md	    Dockerfile	     notebooks	      tests
CITATION.cff	    docs	     pyproject.toml   third-party-programs.txt
CODE_OF_CONDUCT.md  LICENSE	     README.md	      tools
configs		    MANIFEST.in      requirements     tox.ini
In [7]:
!pip install -e . -q
In [8]:
import anomalib
anomalib.__version__
Out[8]:
'0.3.6'

Importing libraries¶

In [9]:
import numpy as np
import matplotlib.pyplot as plt
import os, pprint, yaml, warnings, math, glob, cv2, random, logging
from IPython.display import Image
In [10]:
def warn(*args, **kwargs):
    pass
warnings.warn = warn
warnings.filterwarnings('ignore')
logger = logging.getLogger("anomalib")
In [11]:
import anomalib
from pytorch_lightning import Trainer, seed_everything
from anomalib.config import get_configurable_parameters
from anomalib.data import get_datamodule
from anomalib.models import get_model
from anomalib.utils.callbacks import LoadModelCallback, get_callbacks
from anomalib.utils.loggers import configure_logger, get_experiment_logger
In [12]:
import torch
print("Torch version:", torch.__version__)
print("CUDA version:", torch.version.cuda)
print("GPU availability:", torch.cuda.is_available())
print("Number of GPU devices:", torch.cuda.device_count())      
print("Name of current GPU:", torch.cuda.get_device_name(0))  

Torch version: 1.11.0+cu102
CUDA version: 10.2
GPU availability: True
Number of GPU devices: 1
Name of current GPU: NVIDIA GeForce GTX 1050 Ti

Checking the dataset¶

In [13]:
def list_files(startpath):
    for root, dirs, files in os.walk(startpath):
        level = root.replace(startpath, '').count(os.sep)
        indent = ' ' * 4 * (level)
        print('{}{}/'.format(indent, os.path.basename(root)))
        subindent = ' ' * 4 * (level + 1)
In [14]:
%cd {PROJECT_PATH}/dataset/MVTec/

/home/dennishnf/Desktop/unsupervised-anomaly-detection/dataset/MVTec
In [15]:
!ls

metal_nut
In [16]:
list_files(PROJECT_PATH+"/dataset/MVTec/")

/
metal_nut/
    ground_truth/
        color/
        bent/
        flip/
        scratch/
    test/
        color/
        bent/
        flip/
        good/
        scratch/
    train/
        good/

Models¶

PaDiM: A Patch Distribution Modeling Framework for Anomaly Detection and Localization¶

Paper: PaDiM

PaDiM is a patch based algorithm. It relies on a pre-trained CNN feature extractor. The image is broken into patches and embeddings are extracted from each patch using different layers of the feature extractors. The activation vectors from different layers are concatenated to get embedding vectors carrying information from different semantic levels and resolutions. This helps encode fine grained and global contexts. However, since the generated embedding vectors may carry redundant information, dimensions are reduced using random selection. A multivariate gaussian distribution is generated for each patch embedding across the entire training batch. Thus, for each patch of the set of training images, we have a different multivariate gaussian distribution. These gaussian distributions are represented as a matrix of gaussian parameters.

During inference, Mahalanobis distance is used to score each patch position of the test image. It uses the inverse of the covariance matrix calculated for the patch during training. The matrix of Mahalanobis distances forms the anomaly map with higher scores indicating anomalous regions.

PaDiM Architecture

PatchCore¶

Paper: PatchCore

The PatchCore algorithm is based on the idea that an image can be classified as anomalous as soon as a single patch is anomalous. The input image is tiled. These tiles act as patches which are fed into the neural network. It consists of a single pre-trained network which is used to extract "mid" level features patches. The "mid" level here refers to the feature extraction layer of the neural network model. Lower level features are generally too broad and higher level features are specific to the dataset the model is trained on. The features extracted during training phase are stored in a memory bank of neighbourhood aware patch level features.

During inference this memory bank is coreset subsampled. Coreset subsampling generates a subset which best approximates the structure of the available set and allows for approximate solution finding. This subset helps reduce the search cost associated with nearest neighbour search. The anomaly score is taken as the maximum distance between the test patch in the test patch collection to each respective nearest neighbour.

PatchCore Architecture

Student-Teacher Feature Pyramid Matching for Unsupervised Anomaly Detection¶

Paper: STFPM

STFPM algorithm consists of a pre-trained teacher network and a student network with identical architecture. The student network learns the distribution of anomaly-free images by matching the features with the counterpart features in the teacher network. Multi-scale feature matching is used to enhance robustness. This hierarchical feature matching enables the student network to receive a mixture of multi-level knowledge from the feature pyramid thus allowing for anomaly detection of various sizes.

During inference, the feature pyramids of teacher and student networks are compared. Larger difference indicates a higher probability of anomaly occurrence.

STFPM Architecture

FastFlow: Unsupervised Anomaly Detection and Localization via 2D Normalizing Flows¶

Paper: FastFlow

FastFlow is a two-dimensional normalizing flow-based probability distribution estimator. It can be used as a plug-in module with any deep feature extractor, such as ResNet and vision transformer, for unsupervised anomaly detection and localisation. In the training phase, FastFlow learns to transform the input visual feature into a tractable distribution, and in the inference phase, it assesses the likelihood of identifying anomalies.

FastFlow Architecture

Anomaly Detection via Reverse Distillation from One-Class Embedding¶

Paper: Reverse Distillation

Reverse Distillation model consists of three networks. The first is a pre-trained feature extractor (E). The next two are the one-class bottleneck embedding (OCBE) and the student decoder network (D). The backbone E is a ResNet model pre-trained on ImageNet dataset. During the forward pass, features from three ResNet block are extracted. These features are encoded by concatenating the three feature maps using the multi-scale feature fusion block of OCBE and passed to the decoder D. The decoder network is symmetrical to the feature extractor but reversed. During training, outputs from these symmetrical blocks are forced to be similar to the corresponding feature extractor layers by using cosine distance as the loss metric.

During testing, a similar step is followed but this time the cosine distance between the feature maps is used to indicate the presence of anomalies. The distance maps from all the three layers are up-sampled to the image size and added (or multiplied) to produce the final feature map. Gaussian blur is applied to the output map to make it smoother. Finally, the anomaly map is generated by applying min-max normalization on the output map.

Anomaly Detection via Reverse Distillation from One-Class Embedding Architecture

Setting config files¶

In [17]:
%cd {PROJECT_PATH}

/home/dennishnf/Desktop/unsupervised-anomaly-detection
In [18]:
!ls

anomalib   unsupervised-anomaly-detection.html
dataset    unsupervised-anomaly-detection.ipynb
README.md
In [19]:
CONFIG_PATHS = PROJECT_PATH + '/anomalib/anomalib/models'
MODEL_CONFIG_PAIRS = {
    'patchcore': f'{CONFIG_PATHS}/patchcore/config.yaml',
    'padim':     f'{CONFIG_PATHS}/padim/config.yaml',
    'cflow':     f'{CONFIG_PATHS}/cflow/config.yaml',
    'dfkde':     f'{CONFIG_PATHS}/dfkde/config.yaml',
    'dfm':       f'{CONFIG_PATHS}/dfm/config.yaml',
    'ganomaly':  f'{CONFIG_PATHS}/ganomaly/config.yaml',
    'stfpm':     f'{CONFIG_PATHS}/stfpm/config.yaml',
    'fastflow':  f'{CONFIG_PATHS}/fastflow/config.yaml',
    'draem':     f'{CONFIG_PATHS}/draem/config.yaml',
    'reverse_distillation': f'{CONFIG_PATHS}/reverse_distillation/config.yaml',
}
In [20]:
MODEL = 'reverse_distillation'
print(open(os.path.join(MODEL_CONFIG_PAIRS[MODEL]), 'r').read())

dataset:
  name: mvtec #options: [mvtec, btech, folder]
  format: mvtec
  path: ./datasets/MVTec
  category: bottle
  task: segmentation
  image_size: 256
  train_batch_size: 32
  test_batch_size: 32
  inference_batch_size: 32
  num_workers: 8
  transform_config:
    train: null
    val: null
  create_validation_set: false
  tiling:
    apply: false
    tile_size: 64
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

model:
  name: reverse_distillation
  lr: 0.005
  backbone: wide_resnet50_2
  pre_trained: true
  layers:
    - layer1
    - layer2
    - layer3
  early_stopping:
    patience: 3
    metric: pixel_AUROC
    mode: max
  beta1: 0.5
  beta2: 0.99
  normalization_method: min_max # options: [null, min_max, cdf]
  anomaly_map_mode: multiply

metrics:
  image:
    - F1Score
    - AUROC
  pixel:
    - F1Score
    - AUROC
  threshold:
    image_default: 0
    pixel_default: 0
    adaptive: true

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: ./results

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: null #options: onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  accumulate_grad_batches: 1
  amp_backend: native
  auto_lr_find: false
  auto_scale_batch_size: false
  auto_select_gpus: false
  benchmark: false
  check_val_every_n_epoch: 2
  default_root_dir: null
  detect_anomaly: false
  deterministic: false
  devices: 1
  enable_checkpointing: true
  enable_model_summary: true
  enable_progress_bar: true
  fast_dev_run: false
  gpus: null # Set automatically
  gradient_clip_val: 0
  ipus: null
  limit_predict_batches: 1.0
  limit_test_batches: 1.0
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  log_every_n_steps: 50
  log_gpu_memory: null
  max_epochs: 200
  max_steps: null
  min_epochs: null
  min_steps: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle
  num_nodes: 1
  num_processes: null
  num_sanity_val_steps: 0
  overfit_batches: 0.0
  plugins: null
  precision: 32
  profiler: null
  reload_dataloaders_every_n_epochs: 0
  replace_sampler_ddp: true
  strategy: null
  sync_batchnorm: false
  tpu_cores: null
  track_grad_norm: -1
  val_check_interval: 1.0
In [21]:
new_update = {
    "path": PROJECT_PATH + '/dataset/MVTec',
    'task': 'segmentation',
    'category': 'metal_nut', 
    'image_size': 256,
    'train_batch_size': 4,
    'test_batch_size': 4,
    'max_epochs': 4,
    'seed': 101
}
In [22]:
# update yaml key's value
def update_yaml(old_yaml, new_yaml, new_update):
    # load yaml
    with open(old_yaml) as f:
        old = yaml.safe_load(f)
                  
    temp = []
    def set_state(old, key, value):
        if isinstance(old, dict):
            for k, v in old.items():
                if k == 'project':
                    temp.append(k)
                if k == key:
                    if temp and k == 'path':
                        # right now, we don't wanna change `project.path`
                        continue
                    old[k] = value
                elif isinstance(v, dict):
                    set_state(v, key, value)
    
    # iterate over the new update key-value pari
    for key, value in new_update.items():
        set_state(old, key, value)
    
    # save the updated / modified yaml file
    with open(new_yaml, 'w') as f:
        yaml.safe_dump(old, f, default_flow_style=False)
In [23]:
# let's set a new path location of new config file 
new_yaml_path = CONFIG_PATHS + '/' + MODEL + '_new.yaml'
In [24]:
new_yaml_path
Out[24]:
'/home/dennishnf/Desktop/unsupervised-anomaly-detection/anomalib/anomalib/models/reverse_distillation_new.yaml'
In [25]:
# run the update yaml method to update desired key's values
update_yaml(MODEL_CONFIG_PAIRS[MODEL], new_yaml_path, new_update)
In [26]:
with open(new_yaml_path) as f:
    updated_config = yaml.safe_load(f)
pprint.pprint(updated_config) # check if it's updated

{'dataset': {'category': 'metal_nut',
             'create_validation_set': False,
             'format': 'mvtec',
             'image_size': 256,
             'inference_batch_size': 32,
             'name': 'mvtec',
             'num_workers': 8,
             'path': '/home/dennishnf/Desktop/unsupervised-anomaly-detection/dataset/MVTec',
             'task': 'segmentation',
             'test_batch_size': 4,
             'tiling': {'apply': False,
                        'random_tile_count': 16,
                        'remove_border_count': 0,
                        'stride': None,
                        'tile_size': 64,
                        'use_random_tiling': False},
             'train_batch_size': 4,
             'transform_config': {'train': None, 'val': None}},
 'logging': {'log_graph': False, 'logger': []},
 'metrics': {'image': ['F1Score', 'AUROC'],
             'pixel': ['F1Score', 'AUROC'],
             'threshold': {'adaptive': True,
                           'image_default': 0,
                           'pixel_default': 0}},
 'model': {'anomaly_map_mode': 'multiply',
           'backbone': 'wide_resnet50_2',
           'beta1': 0.5,
           'beta2': 0.99,
           'early_stopping': {'metric': 'pixel_AUROC',
                              'mode': 'max',
                              'patience': 3},
           'layers': ['layer1', 'layer2', 'layer3'],
           'lr': 0.005,
           'name': 'reverse_distillation',
           'normalization_method': 'min_max',
           'pre_trained': True},
 'optimization': {'export_mode': None},
 'project': {'path': './results', 'seed': 101},
 'trainer': {'accelerator': 'auto',
             'accumulate_grad_batches': 1,
             'amp_backend': 'native',
             'auto_lr_find': False,
             'auto_scale_batch_size': False,
             'auto_select_gpus': False,
             'benchmark': False,
             'check_val_every_n_epoch': 2,
             'default_root_dir': None,
             'detect_anomaly': False,
             'deterministic': False,
             'devices': 1,
             'enable_checkpointing': True,
             'enable_model_summary': True,
             'enable_progress_bar': True,
             'fast_dev_run': False,
             'gpus': None,
             'gradient_clip_val': 0,
             'ipus': None,
             'limit_predict_batches': 1.0,
             'limit_test_batches': 1.0,
             'limit_train_batches': 1.0,
             'limit_val_batches': 1.0,
             'log_every_n_steps': 50,
             'log_gpu_memory': None,
             'max_epochs': 4,
             'max_steps': None,
             'min_epochs': None,
             'min_steps': None,
             'move_metrics_to_cpu': False,
             'multiple_trainloader_mode': 'max_size_cycle',
             'num_nodes': 1,
             'num_processes': None,
             'num_sanity_val_steps': 0,
             'overfit_batches': 0.0,
             'plugins': None,
             'precision': 32,
             'profiler': None,
             'reload_dataloaders_every_n_epochs': 0,
             'replace_sampler_ddp': True,
             'strategy': None,
             'sync_batchnorm': False,
             'tpu_cores': None,
             'track_grad_norm': -1,
             'val_check_interval': 1.0},
 'visualization': {'image_save_path': None,
                   'log_images': True,
                   'mode': 'full',
                   'save_images': True,
                   'show_images': False}}

Training¶

In [27]:
if updated_config['project']['seed'] != 0:
    print(updated_config['project']['seed'])
    seed_everything(updated_config['project']['seed'])

Global seed set to 101

101
In [28]:
# It will return the configurable parameters in DictConfig object.
config = get_configurable_parameters(
    model_name=updated_config['model']['name'],
    config_path=new_yaml_path
)
In [29]:
# pass the config file to model, logger, callbacks and datamodule
model      = get_model(config)
experiment_logger = get_experiment_logger(config)
callbacks  = get_callbacks(config)
datamodule = get_datamodule(config)

Transform configs has not been provided. Images will be normalized using ImageNet statistics.
Transform configs has not been provided. Images will be normalized using ImageNet statistics.
In [30]:
# start training
trainer = Trainer(**config.trainer, logger=experiment_logger, callbacks=callbacks)
trainer.fit(model=model, datamodule=datamodule)

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
`Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..
`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..
`Trainer(limit_predict_batches=1.0)` was configured so 100% of the batches will be used..
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name                  | Type                     | Params
-------------------------------------------------------------------
0 | image_threshold       | AdaptiveThreshold        | 0     
1 | pixel_threshold       | AdaptiveThreshold        | 0     
2 | model                 | ReverseDistillationModel | 80.6 M
3 | image_metrics         | AnomalibMetricCollection | 0     
4 | pixel_metrics         | AnomalibMetricCollection | 0     
5 | normalization_metrics | MinMax                   | 0     
-------------------------------------------------------------------
80.6 M    Trainable params
0         Non-trainable params
80.6 M    Total params
322.438   Total estimated model params size (MB)

Epoch 1:  65%|██████████████▍       | 55/84 [00:30<00:16,  1.79it/s, loss=0.327]
Validation: 0it [00:00, ?it/s]
Validation:   0%|                                        | 0/29 [00:00<?, ?it/s]
Validation DataLoader 0:   0%|                           | 0/29 [00:00<?, ?it/s]
Validation DataLoader 0:   3%|▋                  | 1/29 [00:00<00:09,  3.00it/s]
Epoch 1:  67%|██████████████▋       | 56/84 [00:32<00:16,  1.72it/s, loss=0.327]
Validation DataLoader 0:   7%|█▎                 | 2/29 [00:00<00:07,  3.64it/s]
Epoch 1:  68%|██████████████▉       | 57/84 [00:32<00:15,  1.74it/s, loss=0.327]
Validation DataLoader 0:  10%|█▉                 | 3/29 [00:00<00:06,  3.90it/s]
Epoch 1:  69%|███████████████▏      | 58/84 [00:32<00:14,  1.76it/s, loss=0.327]
Validation DataLoader 0:  14%|██▌                | 4/29 [00:01<00:05,  4.21it/s]
Epoch 1:  70%|███████████████▍      | 59/84 [00:33<00:14,  1.78it/s, loss=0.327]
Validation DataLoader 0:  17%|███▎               | 5/29 [00:01<00:05,  4.53it/s]
Epoch 1:  71%|███████████████▋      | 60/84 [00:33<00:13,  1.80it/s, loss=0.327]
Validation DataLoader 0:  21%|███▉               | 6/29 [00:01<00:04,  4.68it/s]
Epoch 1:  73%|███████████████▉      | 61/84 [00:33<00:12,  1.82it/s, loss=0.327]
Validation DataLoader 0:  24%|████▌              | 7/29 [00:01<00:04,  4.81it/s]
Epoch 1:  74%|████████████████▏     | 62/84 [00:33<00:11,  1.84it/s, loss=0.327]
Validation DataLoader 0:  28%|█████▏             | 8/29 [00:01<00:04,  4.92it/s]
Epoch 1:  75%|████████████████▌     | 63/84 [00:33<00:11,  1.86it/s, loss=0.327]
Validation DataLoader 0:  31%|█████▉             | 9/29 [00:02<00:04,  4.83it/s]
Epoch 1:  76%|████████████████▊     | 64/84 [00:34<00:10,  1.87it/s, loss=0.327]
Validation DataLoader 0:  34%|██████▏           | 10/29 [00:02<00:03,  4.95it/s]
Epoch 1:  77%|█████████████████     | 65/84 [00:34<00:10,  1.89it/s, loss=0.327]
Validation DataLoader 0:  38%|██████▊           | 11/29 [00:02<00:03,  4.88it/s]
Epoch 1:  79%|█████████████████▎    | 66/84 [00:34<00:09,  1.91it/s, loss=0.327]
Validation DataLoader 0:  41%|███████▍          | 12/29 [00:02<00:03,  4.81it/s]
Epoch 1:  80%|█████████████████▌    | 67/84 [00:34<00:08,  1.93it/s, loss=0.327]
Validation DataLoader 0:  45%|████████          | 13/29 [00:02<00:03,  4.85it/s]
Epoch 1:  81%|█████████████████▊    | 68/84 [00:35<00:08,  1.94it/s, loss=0.327]
Validation DataLoader 0:  48%|████████▋         | 14/29 [00:03<00:03,  4.89it/s]
Epoch 1:  82%|██████████████████    | 69/84 [00:35<00:07,  1.96it/s, loss=0.327]
Validation DataLoader 0:  52%|█████████▎        | 15/29 [00:03<00:02,  5.05it/s]
Epoch 1:  83%|██████████████████▎   | 70/84 [00:35<00:07,  1.98it/s, loss=0.327]
Validation DataLoader 0:  55%|█████████▉        | 16/29 [00:03<00:02,  5.18it/s]
Epoch 1:  85%|██████████████████▌   | 71/84 [00:35<00:06,  2.00it/s, loss=0.327]
Validation DataLoader 0:  59%|██████████▌       | 17/29 [00:03<00:02,  5.32it/s]
Epoch 1:  86%|██████████████████▊   | 72/84 [00:35<00:05,  2.02it/s, loss=0.327]
Validation DataLoader 0:  62%|███████████▏      | 18/29 [00:03<00:02,  5.43it/s]
Epoch 1:  87%|███████████████████   | 73/84 [00:35<00:05,  2.03it/s, loss=0.327]
Validation DataLoader 0:  66%|███████████▊      | 19/29 [00:03<00:01,  5.55it/s]
Epoch 1:  88%|███████████████████▍  | 74/84 [00:36<00:04,  2.05it/s, loss=0.327]
Validation DataLoader 0:  69%|████████████▍     | 20/29 [00:04<00:01,  5.62it/s]
Epoch 1:  89%|███████████████████▋  | 75/84 [00:36<00:04,  2.07it/s, loss=0.327]
Validation DataLoader 0:  72%|█████████████     | 21/29 [00:04<00:01,  5.70it/s]
Epoch 1:  90%|███████████████████▉  | 76/84 [00:36<00:03,  2.09it/s, loss=0.327]
Validation DataLoader 0:  76%|█████████████▋    | 22/29 [00:04<00:01,  5.66it/s]
Epoch 1:  92%|████████████████████▏ | 77/84 [00:36<00:03,  2.10it/s, loss=0.327]
Validation DataLoader 0:  79%|██████████████▎   | 23/29 [00:04<00:01,  5.59it/s]
Epoch 1:  93%|████████████████████▍ | 78/84 [00:36<00:02,  2.12it/s, loss=0.327]
Validation DataLoader 0:  83%|██████████████▉   | 24/29 [00:04<00:00,  5.53it/s]
Epoch 1:  94%|████████████████████▋ | 79/84 [00:36<00:02,  2.14it/s, loss=0.327]
Validation DataLoader 0:  86%|███████████████▌  | 25/29 [00:04<00:00,  5.56it/s]
Epoch 1:  95%|████████████████████▉ | 80/84 [00:37<00:01,  2.15it/s, loss=0.327]
Validation DataLoader 0:  90%|████████████████▏ | 26/29 [00:05<00:00,  5.65it/s]
Epoch 1:  96%|█████████████████████▏| 81/84 [00:37<00:01,  2.17it/s, loss=0.327]
Validation DataLoader 0:  93%|████████████████▊ | 27/29 [00:05<00:00,  5.69it/s]
Epoch 1:  98%|█████████████████████▍| 82/84 [00:37<00:00,  2.19it/s, loss=0.327]
Validation DataLoader 0:  97%|█████████████████▍| 28/29 [00:05<00:00,  5.75it/s]
Epoch 1:  99%|█████████████████████▋| 83/84 [00:37<00:00,  2.20it/s, loss=0.327]
Validation DataLoader 0: 100%|██████████████████| 29/29 [00:05<00:00,  6.10it/s]
Epoch 1: 100%|█| 84/84 [00:41<00:00,  2.05it/s, loss=0.327, pixel_F1Score=0.770,
Epoch 3:  65%|▋| 55/84 [00:38<00:20,  1.42it/s, loss=0.246, pixel_F1Score=0.770,
Validation: 0it [00:00, ?it/s]
Validation:   0%|                                        | 0/29 [00:00<?, ?it/s]
Validation DataLoader 0:   0%|                           | 0/29 [00:00<?, ?it/s]
Validation DataLoader 0:   3%|▋                  | 1/29 [00:00<00:14,  1.91it/s]
Epoch 3:  67%|▋| 56/84 [00:44<00:22,  1.26it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:   7%|█▎                 | 2/29 [00:00<00:10,  2.50it/s]
Epoch 3:  68%|▋| 57/84 [00:44<00:21,  1.27it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  10%|█▉                 | 3/29 [00:01<00:08,  2.98it/s]
Epoch 3:  69%|▋| 58/84 [00:45<00:20,  1.29it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  14%|██▌                | 4/29 [00:01<00:07,  3.45it/s]
Epoch 3:  70%|▋| 59/84 [00:45<00:19,  1.30it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  17%|███▎               | 5/29 [00:01<00:06,  3.78it/s]
Epoch 3:  71%|▋| 60/84 [00:45<00:18,  1.32it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  21%|███▉               | 6/29 [00:01<00:05,  4.03it/s]
Epoch 3:  73%|▋| 61/84 [00:45<00:17,  1.33it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  24%|████▌              | 7/29 [00:01<00:05,  4.13it/s]
Epoch 3:  74%|▋| 62/84 [00:45<00:16,  1.35it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  28%|█████▏             | 8/29 [00:02<00:04,  4.47it/s]
Epoch 3:  75%|▊| 63/84 [00:46<00:15,  1.37it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  31%|█████▉             | 9/29 [00:02<00:04,  4.52it/s]
Epoch 3:  76%|▊| 64/84 [00:46<00:14,  1.38it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  34%|██████▏           | 10/29 [00:02<00:04,  4.60it/s]
Epoch 3:  77%|▊| 65/84 [00:46<00:13,  1.40it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  38%|██████▊           | 11/29 [00:02<00:03,  4.70it/s]
Epoch 3:  79%|▊| 66/84 [00:46<00:12,  1.41it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  41%|███████▍          | 12/29 [00:03<00:03,  4.72it/s]
Epoch 3:  80%|▊| 67/84 [00:46<00:11,  1.43it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  45%|████████          | 13/29 [00:03<00:03,  4.80it/s]
Epoch 3:  81%|▊| 68/84 [00:47<00:11,  1.44it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  48%|████████▋         | 14/29 [00:03<00:03,  4.93it/s]
Epoch 3:  82%|▊| 69/84 [00:47<00:10,  1.46it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  52%|█████████▎        | 15/29 [00:03<00:02,  5.14it/s]
Epoch 3:  83%|▊| 70/84 [00:47<00:09,  1.47it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  55%|█████████▉        | 16/29 [00:03<00:02,  5.26it/s]
Epoch 3:  85%|▊| 71/84 [00:47<00:08,  1.49it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  59%|██████████▌       | 17/29 [00:03<00:02,  5.25it/s]
Epoch 3:  86%|▊| 72/84 [00:47<00:07,  1.50it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  62%|███████████▏      | 18/29 [00:04<00:02,  5.26it/s]
Epoch 3:  87%|▊| 73/84 [00:48<00:07,  1.52it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  66%|███████████▊      | 19/29 [00:04<00:01,  5.44it/s]
Epoch 3:  88%|▉| 74/84 [00:48<00:06,  1.53it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  69%|████████████▍     | 20/29 [00:04<00:01,  5.48it/s]
Epoch 3:  89%|▉| 75/84 [00:48<00:05,  1.55it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  72%|█████████████     | 21/29 [00:04<00:01,  5.60it/s]
Epoch 3:  90%|▉| 76/84 [00:48<00:05,  1.56it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  76%|█████████████▋    | 22/29 [00:04<00:01,  5.65it/s]
Epoch 3:  92%|▉| 77/84 [00:48<00:04,  1.58it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  79%|██████████████▎   | 23/29 [00:04<00:01,  5.68it/s]
Epoch 3:  93%|▉| 78/84 [00:48<00:03,  1.59it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  83%|██████████████▉   | 24/29 [00:05<00:00,  5.70it/s]
Epoch 3:  94%|▉| 79/84 [00:49<00:03,  1.61it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  86%|███████████████▌  | 25/29 [00:05<00:00,  5.70it/s]
Epoch 3:  95%|▉| 80/84 [00:49<00:02,  1.62it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  90%|████████████████▏ | 26/29 [00:05<00:00,  5.71it/s]
Epoch 3:  96%|▉| 81/84 [00:49<00:01,  1.64it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  93%|████████████████▊ | 27/29 [00:05<00:00,  5.69it/s]
Epoch 3:  98%|▉| 82/84 [00:49<00:01,  1.65it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0:  97%|█████████████████▍| 28/29 [00:05<00:00,  5.59it/s]
Epoch 3:  99%|▉| 83/84 [00:49<00:00,  1.67it/s, loss=0.246, pixel_F1Score=0.770,
Validation DataLoader 0: 100%|██████████████████| 29/29 [00:06<00:00,  5.79it/s]
Epoch 3: 100%|█| 84/84 [00:52<00:00,  1.59it/s, loss=0.246, pixel_F1Score=0.802,
Epoch 3: 100%|█| 84/84 [01:03<00:00,  1.31it/s, loss=0.246, pixel_F1Score=0.802,

Evaluation¶

In [31]:
# load best model from checkpoint before evaluating
load_model_callback = LoadModelCallback(
    weights_path=trainer.checkpoint_callback.best_model_path
)
trainer.callbacks.insert(0, load_model_callback)
trainer.test(model=model, datamodule=datamodule)

The following callbacks returned in `LightningModule.configure_callbacks` will override existing callbacks passed to Trainer: EarlyStopping
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

Testing DataLoader 0: 100%|█████████████████████| 29/29 [00:32<00:00,  1.12s/it]
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       Test metric             DataLoader 0
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       image_AUROC          0.5830889344215393
      image_F1Score         0.8942307829856873
       pixel_AUROC          0.9722244739532471
      pixel_F1Score         0.8017732501029968
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Out[31]:
[{'pixel_F1Score': 0.8017732501029968,
  'pixel_AUROC': 0.9722244739532471,
  'image_F1Score': 0.8942307829856873,
  'image_AUROC': 0.5830889344215393}]

Visualization of the prediction on the test dataset¶

In [32]:
RESULT_PATH = os.path.join(
    updated_config['project']['path'],
    updated_config['model']['name'],
    updated_config['dataset']['format'], 
    updated_config['dataset']['category']
)
RESULT_PATH
Out[32]:
'./results/reverse_distillation/mvtec/metal_nut'
In [33]:
# a simple function to visualize the model's prediction (anomaly heatmap)
def visualiz(paths, n_images, is_random=True, figsize=(16, 16)):
    for i in range(n_images):
        image_name = paths[i]
        if is_random: image_name = random.choice(paths)
        img = cv2.imread(image_name)[:,:,::-1]
        
        category_type = image_name.split('/')[-4:-3:][0]
        defected_type = image_name.split('/')[-2:-1:][0]
        
        plt.figure(figsize=figsize)
        plt.imshow(img)
        plt.title(
            f"Category : {category_type} and Defected Type : {defected_type} \n {image_name}", 
            fontdict={'fontsize': 20, 'fontweight': 'medium'}
        )
        plt.xticks([])
        plt.yticks([])
        plt.tight_layout()
    plt.show()
In [34]:
for content in os.listdir(RESULT_PATH):
    if content == 'images':
        full_path = glob.glob(os.path.join(RESULT_PATH, content, '**',  '*.png'), recursive=True)
        print('Total Image ', len(full_path))
        print(full_path[0].split('/'))
        print(full_path[0].split('/')[-2:-1:])
        print(full_path[0].split('/')[-4:-3:])

Total Image  117
['.', 'results', 'reverse_distillation', 'mvtec', 'metal_nut', 'images', 'image_ROC.png']
['images']
['mvtec']
In [35]:
visualiz(full_path, 10, is_random=True, figsize=(30, 30))

Inference on new images¶

In [36]:
%cd {PROJECT_PATH}

/home/dennishnf/Desktop/unsupervised-anomaly-detection
In [37]:
infer_resutls = PROJECT_PATH + "/infer_results/"
In [38]:
# anomalies: color, bent, flip, scratch
# images: 1 to ~20
In [39]:
# input image
input_img = PROJECT_PATH + "/dataset/MVTec/metal_nut/test/bent/013.png"
input_img
Out[39]:
'/home/dennishnf/Desktop/unsupervised-anomaly-detection/dataset/MVTec/metal_nut/test/bent/013.png'
In [40]:
# output image
output_img = input_img.replace(PROJECT_PATH + "/dataset/MVTec/metal_nut/test/", infer_resutls)
output_img
Out[40]:
'/home/dennishnf/Desktop/unsupervised-anomaly-detection/infer_results/bent/013.png'
In [41]:
# perform inference on the sample image
!python -W ignore anomalib/tools/inference/lightning_inference.py \
        --config {new_yaml_path} \
        --weights {trainer.checkpoint_callback.best_model_path} \
        --input {input_img} \
        --output infer_results

/home/dennishnf/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py:51: LightningDeprecationWarning: Setting `Trainer(resume_from_checkpoint=)` is deprecated in v1.5 and will be removed in v1.7. Please pass `Trainer.fit(ckpt_path=)` directly instead.
  rank_zero_deprecation(
/home/dennishnf/miniconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py:52: LightningDeprecationWarning: Setting `max_steps = None` is deprecated in v1.5 and will no longer be supported in v1.7. Use `max_steps = -1` instead.
  rank_zero_deprecation(
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
`Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..
`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..
`Trainer(limit_predict_batches=1.0)` was configured so 100% of the batches will be used..
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
Transform configs has not been provided. Images will be normalized using ImageNet statistics.
Missing logger folder: results/reverse_distillation/mvtec/metal_nut/lightning_logs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Predicting DataLoader 0: 100%|████████████████████| 1/1 [00:00<00:00,  7.69it/s]
In [42]:
print(input_img)
display(Image(input_img, width=250))

/home/dennishnf/Desktop/unsupervised-anomaly-detection/dataset/MVTec/metal_nut/test/bent/013.png
In [43]:
print(output_img)
display(Image(output_img, width=250))

/home/dennishnf/Desktop/unsupervised-anomaly-detection/infer_results/bent/013.png

Summary of evaluating some arquitectures¶

MODEL = 'padim'
'image_size': 256,
'train_batch_size': 4,
'test_batch_size': 4,
'max_epochs': 4,
[{'pixel_F1Score': 0.7553240060806274,
  'pixel_AUROC': 0.9686623215675354,
  'image_F1Score': 0.952380895614624,
  'image_AUROC': 0.9496578574180603}]
MODEL = 'patchcore'
'image_size': 128,
'train_batch_size': 1,
'test_batch_size': 1,
'max_epochs': 3,
[{'pixel_F1Score': 0.8109800219535828,
  'pixel_AUROC': 0.9827464818954468,
  'image_F1Score': 0.9726775884628296,
  'image_AUROC': 0.9833822250366211}]
MODEL = 'stfpm'
'image_size': 256,
'train_batch_size': 4,
'test_batch_size': 4,
'max_epochs': 4,
[{'pixel_F1Score': 0.6699215769767761,
  'pixel_AUROC': 0.9740051627159119,
  'image_F1Score': 0.9555555582046509,
  'image_AUROC': 0.9838709235191345}]
MODEL = 'fastflow'
'image_size': 256,
'train_batch_size': 4,
'test_batch_size': 4,
'max_epochs': 4,
[{'pixel_F1Score': 0.7417428493499756,
  'pixel_AUROC': 0.9636613130569458,
  'image_F1Score': 0.9560439586639404,
  'image_AUROC': 0.9310851097106934}]
MODEL = 'reverse_distillation'
'image_size': 256,
'train_batch_size': 4,
'test_batch_size': 4,
'max_epochs': 4,
[{'pixel_F1Score': 0.7951029539108276,
  'pixel_AUROC': 0.980124831199646,
  'image_F1Score': 0.9723756313323975,
  'image_AUROC': 0.9672531485557556}]